Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus combines a partial area of a first image with a second image. The partial area to be combined with the second image is determined based on distance information with regard to a plurality of partial areas of the first image.

BACKGROUND

Field

Technical Field

Aspects of the embodiment(s) relate to an image processing apparatus andan image processing method, and more particularly to image combiningtechniques.

DESCRIPTION OF THE RELATED ART

An apparatus including a function of displaying a composite imagegenerated by combining a portion of a certain image (main image) withanother image (sub image) has been available.

Normally, a sub image is non-transparent. Thus, an area of a main imagecombined with a sub image becomes hidden. Japanese Patent Laid-Open No.2010-103651 proposes to determine an area of a main image to be combinedwith a sub image based on the position of a face area of a persondetected in the main image.

However, Japanese Patent Laid-Open No. 2010-103651 does not disclose howto determine an area to be combined with a sub image when no face of aperson is detected in a main image. Techniques disclosed by JapanesePatent Laid-Open No. 2010-103651 do not address a situation where thebackground area also includes an area undesired to be hidden, like acommemorative photo taken at a tourist spot.

SUMMARY

Embodiments of the invention provide an image processing apparatusincluding one or more processors, and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the image processing apparatus to function as an obtaining unitconfigured to obtain a first image and a second image, a determinationunit configured to determine a partial area of the first image as acomposite area to be combined with the second image, and a combiningunit configured to combine the second image with the composite area. Thedetermination unit determines the composite area based on distanceinformation with regard to a plurality of partial areas of the firstimage.

Further features will become apparent from the following description ofexemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an exemplary functional configurationof a digital camera according to an embodiment.

FIGS. 2A and 2B are flowcharts related to a composite image generatingprocess according to the embodiment.

FIGS. 3A to 3D are diagrams illustrating exemplary settings of candidateareas and a prohibited area according to the embodiment.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, an exemplary embodiment will be described in detail withreference to the accompanying drawings. Although a digital camera thatgenerates a captured image and distance information of a subject will bedescribed below as an example of an image processing apparatus accordingto an embodiment of the invention, configurations for capturing an imageor generating distance information of a subject are not essential to theembodiment of the invention. For example, it is only necessary that animage and distance information of a corresponding subject be obtained,and can be done so using any means, such as obtaining a pre-recordedimage and distance information of a corresponding subject from a storagedevice or the like. Therefore, the embodiment of the invention can beimplemented in any electronic equipment including a personal computer, amobile phone, a game machine, etc.

FIG. 1 is a block diagram illustrating an exemplary functionalconfiguration of a digital camera 100 according to the embodiment of theinvention. The digital camera 100 is a multi-eye camera including a mainimage capturing unit 190 and a sub image capturing unit 191.Alternatively, the digital camera 100 can be a single-eye cameraincluding one image capturing unit. The main image capturing unit 190and the sub image capturing unit 191 each include a barrier 102, animage capturing lens 103, a shutter 101, an image sensor 122, and ananalog-to-digital (A/D) converter 123.

The barrier 102 is a movable member that covers the tip of the imagecapturing lens 103. The barrier 102 protects the image capturing lens103 from external impact or dust mainly when the power is turned off.

The image capturing lens 103 includes a zoom lens and a focus lens, andforms an optical image of a subject on an imaging plane of the imagesensor 122. The shutter 101 includes an aperture function.

The image sensor 122 includes color filters and an array of microlenses,and includes a configuration where a plurality of pixels is arrangedtwo-dimensionally. Each pixel includes one microlens and a plurality ofphotoelectric converters (sub-pixels). Since a partial area of the exitpupil projected to the individual sub-pixels of each pixel is differentfrom one another, a parallax image can be obtained from the individualoutputs of sub-pixel groups with an equal partial area of thecorresponding exit pupil. The image sensor 122 of the sub imagecapturing unit 191 can have a configuration where one photoelectricconverter is provided for one microlens in each pixel.

The A/D converter 123 converts an analog signal output from the imagesensor 122 to a digital signal and outputs the digital signal as imagedata.

In the embodiment, the optical axes of the main image capturing unit 190and the sub image capturing unit 191 are parallel to each other, but theangle of view of the image capturing lens 103 is different between themain image capturing unit 190 and the sub image capturing unit 191. Theimage capturing range can be different between the main image capturingunit 190 and the sub image capturing unit 191. Alternatively, the imagecapturing range can be settable by the user.

An image processor 124 applies certain image processing to image datasupplied from the A/D converter 123 or a memory controller 115. Imageprocessing applied by the image processor 124 includes white balanceadjustment, pixel interpolation, resizing such as size reduction, colorinterpolation (demosaicing), and color conversion. The image processor124 calculates evaluation values used for auto-focus detection (AF)processing and auto-exposure control (AE) processing executed by asystem controller 150, and supplies the system controller 150 with theseevaluation values. The image processor 124 can perform processingregarding coding/decoding of image data. The image processor 124generates distance information (distance map) from a pair of parallaximages. The image processor 124 can distinguish a captured scene basedon, for example, the image colors, luminance, or subject detectionresult.

A subject detector 125 executes a subject detection process on imagedata supplied from the image processor 124 via the memory controller115, and identifies a subject area. The subject detector 125 executessubject detection using any technique of the related art, such aspattern patching. For example, the subject detector 125 can use a methodthat, in the case of detecting a person's face as a subject area,searches for the color of skin or a luminance pattern related to partssuch as the eyes, nose, or mouth included in the face. The subjectdetector 125 outputs the presence/absence or the number of detectedsubject areas and information, such as the position, size, andreliability, with regard to each of the detected subject areas asdetection results to the system controller 150.

A memory 132 stores image data obtained by converting, using the A/Dconverter 123, an analog signal obtained by the image sensor 122 todigital data, and image data for display on a display unit 128. Becausethe memory 132 functions as a buffer when images are successivelycaptured, the memory 132 has a capacity that can store image data, aswell as audio data in the case of a moving image. The memory 132 alsofunctions as a video memory.

A digital-to-analog (D/A) converter 113 converts image data for display,stored in the memory 132, to an analog signal and supplies the displayunit 128 with the analog signal.

The display unit 128 displays an image signal supplied from the D/Aconverter 113. When a moving image is captured and the moving image isdisplayed in real time on the display unit 128, the display unit 128 canfunction as an electronic viewfinder (live viewing function).

A non-volatile memory 156 is an electrically erasable and recordablememory. The non-volatile memory 156 stores constants for the operationof the system controller 150, and programs. The non-volatile memory 156also stores a program for executing various operations, which will bedescribed below.

The system controller 150 includes one or more microprocessors (centralprocessing units (CPUs)) capable of executing programs, and controls theentire digital camera 100. The system controller 150 performs AFprocessing and AE processing using evaluation values supplied from theimage processor 124, and applies control necessary for realizingfunctions in response to instructions from an operation unit 170. Thesystem controller 150 determines a read-out mode of the image sensor122, i.e., whether to read out a pair of parallax images or to performaddition read-out, and sets the determined read-out mode in the mainimage capturing unit 190.

A system memory 152 is a memory that expands constants or variables forthe operation of the system controller 150, or programs read out fromthe non-volatile memory 156. A system timer 153 measures a time used forvarious types of control or the time of a built-in clock.

A mode selection switch 160 is an input device that inputs aninstruction for changing the operation mode of the digital camera 100 tothe system controller 150. Examples of operation modes include a stillimage mode, a moving image mode, and a play mode.

A first shutter switch 162 is turned on when a shutter button 161 isoperated halfway or half-pressed, and generates a first shutter switchsignal SW1 (image capturing preparation instruction). In response to thefirst shutter switch signal SW1, the system controller 150 startsoperations such as AF processing, AE processing, auto-white balance(AWB) processing, electronic flash (EF) (pre-flash) processing, etc.

A second shutter switch 163 is turned on when the shutter button 161 isoperated completely or full-pressed, and generates a second shutterswitch signal SW2 (image capturing instruction). In response to thesecond shutter switch signal SW2, the system controller 150 starts aseries of image capturing operations including reading of a signal fromthe image sensor 122 to writing of image data to a recording medium 104.

Operation members of the operation unit 170 are appropriately allocatedwith functions on a scene-by-scene basis by selecting and operatingvarious function icons displayed on the display unit 128, and theseoperation members function as various function buttons. The functionbuttons include, for example, an end button, a return button, an imagefeed button, a jump button, a depth-of-field preview button, and anattribute change button. For example, when a menu button is pressed, amenu screen where various settings can be made is displayed on thedisplay unit 128. The user can input various instructions using the menuscreen displayed on the display unit 128, a four-direction button(directed up, down, left, and right), and a set button. When the displayunit 128 is a touchscreen, the touchscreen is also included in theoperation unit 170.

A power controller 180 includes a battery detecting circuit, a directcurrent (DC)-to-DC converter, and a switch circuit for switching a blockto be electrically connected, and detects whether a battery is mounted,the battery type, and the remaining capacity of the battery. Based onthe detection results and an instruction from the system controller 150,the power controller 180 controls the DC-DC converter and supplies eachsection, including the recording medium 104, with a necessary voltagefor a necessary period of time.

A power supply unit 130 includes a primary battery such as an alkalinebattery or a lithium battery, a secondary battery such as a NiCdbattery, a NiMH battery, or a Li battery, and an alternating current(AC) adapter.

A recording medium interface (I/F) 118 is an interface with therecording medium 104, such as a memory card or a hard disk.

A power switch 172 is a switch that provides an instruction for turningon/off the power of the digital camera 100. When the power is turned on,the power controller 180 starts supplying a necessary voltage to eachsection of the digital camera 100, and the system controller 150 startscontrolling the digital camera 100.

The recording medium 104 is a recording medium such as a memory card forrecording captured images, and includes a semiconductor memory or thelike.

A communication unit 154 establishes a wireless or wired connection andtransmits/receives video signals and audio signals. The communicationunit 154 is also connectable to a wireless local area network (LAN) orthe Internet. The communication unit 154 transmits images, includinglive images, captured by the image sensor 122 and images recorded in therecording medium 104, and also receives image data or other varioustypes of information from an external device.

An orientation detector 155 detects the orientation of the digitalcamera 100 with respect to the direction of gravitational force. Basedon the orientation detected by the orientation detector 155, it can bedetermined whether an image captured by the image sensor 122 is an imagetaken while the digital camera 100 is in a horizontal or verticalposition. The system controller 150 adds direction information inaccordance with the orientation detected by the orientation detector 155to an image file of an image captured by the image sensor 122, orrotates the image and records the rotated image. An acceleration sensoror a gyro sensor can be used as the orientation detector 155.

Generation of a distance map will now be described. There are norestrictions with regard to the method of generating a distance map, andany known method can be used. A distance map is information thatrepresents a subject distance on a pixel-by-pixel basis, and can be adepth map where a luminance value represents a distance. These can alsobe referred to as a distance image or a depth image.

Since the image sensor 122 can obtain a pair of parallax images in theembodiment, a subject distance can be obtained on a pixel-by-pixel basisusing techniques such as the stereo matching. Alternatively, mapping canbe done in the form of each area's image displacement amount (parallaxamount) obtained using a pair of parallax images or a defocus amountconverted using a coefficient (K value) determined from the imagedisplacement amount in accordance with an optical condition.

The defocus amount or the image displacement amount itself is not avalue that indicates the absolute value of a subject distance. However,a map based on the defocus amount or the image displacement amountrepresents the relationship of the relative distance of a subject in animage. Only detection of a relative relationship of a subject distanceis sufficient for implementing the embodiment, and the absolute value ofa subject distance is not necessarily required. Therefore, a map basedon the defocus amount or the image displacement amount can be used.

Alternatively, a distance map can be generated without using parallaximages. For example, a subject distance on a pixel-by-pixel basis can beobtained by calculating a focus lens position where a contrastevaluation value becomes maximum on a pixel-by-pixel basis.Alternatively, distance information on a pixel-by-pixel basis can beobtained based on the corresponding relationship between blurring anddistance from image data obtained by capturing the same scene aplurality of times while changing a focal distance and a point spreadfunction (PSF) of the optical system. These techniques are described in,for example, Japanese Patent Laid-Open No. 2010-177741 and U.S. Pat. No.4,965,840.

The digital camera 100 according to the embodiment generates a distancemap when a main image is captured. When the main image is a movingimage, a distance map can be generated on a frame-by-frame basis, or adistance map can be generated for a still image obtained by capturing astill image while capturing a moving image. The generated distance mapcan be recorded in association with a corresponding image in therecording medium 104.

With regard to an image for which a distance map will be generated, thesystem controller 150 applies control to read a pair of parallax imagesfrom the image sensor 122 and, using the pair of parallax images, theimage processor 124 generates a distance map. After generating thedistance map, the image processor 124 generates a to-be-recorded imageby adding the pair of parallax images and stores the distance map andthe to-be-recorded image in association with each other in the memory132.

The operation in the still image mode and the moving image mode of thedigital camera 100 according to the embodiment will be described nextwith reference to the flowcharts illustrated in FIGS. 2A and 2B. Tosimplify the description and understanding, it is assumed that acomposite image will be generated both in the still image mode and themoving image mode. However, the user can be enabled to select whether togenerate and record a composite image or to record only one or both of amain image and a sub image without generating a composite image.

Combining Operation in Still Image Mode

A composite image generating operation in the still image mode will nowbe described with reference to FIG. 2A.

In S200, the system controller 150 obtains a main image (first image)and a distance map associated with the main image. The main image andthe distance map can be obtained by image capturing performed using themain image capturing unit 190, or can be obtained from the recordingmedium 104.

In S201, the system controller 150 obtains a sub image (second image).The sub image can be obtained by image capturing performed using the subimage capturing unit 191, or can be obtained from the recording medium104. The sub image is smaller, i.e., the number of pixels is less, thanthe main image since settings have been made that way at the time ofimage capturing or image processing has been done that way aftercapturing the images. Alternatively, an area usable as a sub image canbe specified by the user through the operation unit 170. In this case,to simplify the understanding and description, it is assumed that thesub image has a predetermined size that is sufficiently smaller than themain image.

In S202, the system controller 150 sets candidate areas and a prohibitedarea in the main image, which will be discussed in more detail below. InS203, the system controller 150 calculates an evaluation value for acandidate area selected based on the candidate areas and the prohibitedarea set in S202, which will be discussed in more detail below.

In S204, the system controller 150 determines whether there is anycandidate area for which no evaluation value has been calculated, and,if it is determined that there is a candidate area for which noevaluation value has been calculated, the process returns to S202.Otherwise, the process proceeds to S205.

In S205, the system controller 150 determines a composite area fromamong the candidate areas based on the evaluation values calculated forthese candidate areas by the processing in S202 to S204. For example,the system controller 150 can determine an area with the lowestevaluation value as a composite area.

In S206, the image processor 124 generates a composite image bycombining the sub image obtained in S201 with the main image obtained inS200 based on the composite area determined in S205. When the sizes ofthe sub image and the composite area are different, the image processor124 resizes the sub image to be less than or equal to the size of thecomposite area, and then combines the main image and the resized subimage.

In S207, the image processor 124 applies recording processing to thecomposite image generated in S206. For example, the recording processingincludes file header generation and coding. The image processor 124records an image file including the composite image in the recordingmedium 104 via the memory controller 115 and the recording medium I/F118.

Candidate Area and Prohibited Area Setting Processing

The details of the candidate area and prohibited area setting processingperformed in S202 of FIG. 2A will now be described. FIG. 3A illustratesan exemplary main image. FIG. 3B illustrates exemplary candidate areas300 set in the main image. The system controller 150 sets a plurality ofpartial areas of the entire main image as candidate areas while changingthe position and size of the individual candidate areas step by step.FIG. 3B schematically illustrates an example in which candidate areas,each having one of three sizes, are set in each of areas obtained bydividing the main image in the horizontal direction and the verticaldirection. In this example, the position of each candidate area isdetermined such that the center of the candidate area is identical tothe center of a corresponding one of the divisional areas.

FIG. 3C is a diagram illustrating exemplary settings of candidate areasand a prohibited area. Like FIG. 3B, after the candidate areas 300 areset, a partial area of the image, which is the central area in thiscase, is set as a prohibited area 301. Control is applied to exclude acandidate area (candidate area 300′ in FIG. 3B) that overlaps theprohibited area 301 from becoming the target of evaluation valuecalculation. In doing so, the central area of the image, which isgenerally highly likely to include a subject that the user wants to takea picture of, is controlled not to be a composite area. A prohibitedarea not based on subject detection is not limited to the central areaof an image. For example, like the sky area of an image captured in anevening scene mode, an area that is highly likely to be important andthat has a more or less determined position can be set as a prohibitedarea.

FIG. 3D is a diagram illustrating exemplary settings of a prohibitedarea different from FIG. 3C. In this example, subject areas 302 detectedby the subject detector 125 and their surrounding area are set as aprohibited area 301. By applying control as in FIG. 3C, an area of aperson who is a subject highly likely to be one that the user wants totake a picture of, or an area near the face of a person is controllednot to be a composite area.

Evaluation Value Calculation Processing

The details of the evaluation value calculation processing performed inS203 of FIG. 2A will now be described. Among the candidate areas set inS202, those that do not overlap the prohibited area are individuallysubjected to the following processing. That is, the image processor 124successively regards each of the pixels of each such area as the pixelof interest and calculates the sum of differences in luminance valuebetween the pixel of interest and eight pixels around or near the pixelof interest. After calculating the total sum of the sums of luminancedifferences for all the pixels in the candidate area, the imageprocessor 124 normalizes the total sum by the number of pixels, therebycalculating a luminance evaluation value Ey.

Specifically, when the size of the candidate area is s pixels in thehorizontal direction and t pixels in the vertical direction, thecoordinates of the pixel of interest are (x, y), and the luminance atthe coordinates (x, y) is L(x, y), the luminance evaluation value Ey canbe obtained by the following equation:

${Ey} = \frac{\sum\limits_{y = 0}^{t - I}{\sum\limits_{x = 0}^{s - I}\left( {\sum\limits_{m = {y - I}}^{y + I}{\sum\limits_{n = {x - I}}^{x + I}{{{L\left( {x,y} \right)} - {L\left( {n,m} \right)}}}}} \right)}}{s \cdot t}$

Like the luminance, for color differences u and V and a distance, theimage processor 124 calculates color evaluation values Eu and Ev and adistance evaluation value Ed using similar methods. Note that theevaluation values may be obtained using other methods. For example, theluminance evaluation value Ey can be the reciprocal of the luminancemaximum value or the luminance average value of pixels in the candidatearea. The distance evaluation value Ed can be the minimum value or theaverage value (or the reciprocal thereof) of subject distancescorresponding to pixels in the candidate area.

The image processor 124 calculates a final evaluation value E from theluminance value Ey, the color evaluation values Eu and Ev, and thedistance evaluation value Ed using the following equation:

E=Wy×Ey+Wu×Eu+Wv×Ev+Wd×Ed

In the above equation, Wy is a weight for the luminance evaluation valueEy. Likewise, Wu, Wv, and Wd are weights for the respective evaluationvalues. The system controller 150 controls a candidate area serving as acomposite area in accordance with the circumstances by changing theweights in accordance with the image capturing mode or the capturedscene. By setting one or more weights to zero, control can be applied todetermine a composite area while specializing in a particular parameteramong luminance, color, and distance.

For example, the evaluation value E based on differences between thepixel of interest and the surrounding pixels basically becomes smalleras the variation in brightness, color, and subject distance among thepixels in the candidate area becomes smaller. Therefore, the evaluationvalue E becomes smaller in an area like blue sky, and the evaluationvalue E becomes greater in an area where there are subjects with variouscolors at various distances, like cityscape.

In general image capturing, an area such as sky often has a lowimportance level, except for the mode of capturing an image of starlitsky. Therefore, according to the embodiment, when a commemorative photois taken in front of a famous building or the like, not only a facearea, but also a building area is difficult to be a composite area, andan area, like sky, where the color or luminance is relatively even islikely to be a composite area. By specializing in the distanceevaluation value, when the subject distance represented by theevaluation value Ed is short or the image displacement amount (parallax)is small, the weight Wd can be made greater in order that the finalevaluation value E be made greater and the candidate area be less likelyto be selected.

When the distance evaluation value Ed is the reciprocal of the minimumvalue or the average value, the weights can be determined such that anarea where the luminance evaluation value Ey and the distance evaluationvalue Ed are small is likely to be determined as a composite area. Acandidate area where the evaluation value E is maximum or a candidatearea where the evaluation value E is minimum can be selected inaccordance with the methods of calculating the evaluation values Ey, Eu,Ev, and Ed.

In this way, an evaluation value regarding one or more of luminanceinformation, color information, and distance information is obtained,and a candidate area that has an evaluation value that best matches apredefined condition under which it is considered that the importancelevel is low is determined as a composite area. Therefore, a compositearea can be determined even when no subject area is detected, and, withregard to areas other than a subject area, it is more likely that anarea whose estimated importance level is low will be combined with a subimage.

Combining Operation in Moving Image Mode

Like the above-described operation in the still image mode, a compositeimage can also be generated in the moving image mode. A composite imagegenerating operation in the moving image mode will now be described withreference to FIG. 2B. Steps for performing the same operation as that inthe still image mode are assigned the same reference numerals, anddescriptions thereof are omitted.

In the moving image mode, it is assumed that both the main imagecapturing unit 190 and the sub image capturing unit 191 are capturing amoving image, or both a main moving image and a sub moving image arebeing reproduced from the recording medium 104. The reproducingprocessing is executed by the image processor 124.

In S208, the system controller 150 obtains a main frame and a distancemap associated with the main frame. The main frame and the distance mapcan be obtained from the image processor 124. The main frame is a frameimage of a main image.

In S209, the system controller 150 obtains a sub frame. The sub framecan be obtained from the image processor 124. The sub frame is a frameimage of a sub image. In steps S202 to S205, the same processing as thatin the still image mode is performed.

In S214, the image processor 124 generates a composite frame bycombining the sub frame obtained in S209 with the main frame obtained inS208 based on the composite area determined in S205. When the sizes ofthe sub frame and the composite area are different, the image processor124 resizes the sub frame to be less than or equal to the size of thecomposite area, and then combines the main frame and the resized subframe.

In S215, the system controller 150 determines whether an end instructionhas been input through the operation unit 170. When the systemcontroller 150 determines that an end instruction has been input, theprocess proceeds to S216. Otherwise, the process returns to S208, andthe system controller 150 executes similar processing on the next frameof the main image and of the sub image.

The processing in S202 to S205 can be executed every few frames of themoving image instead of for each frame. In this case, for frames forwhich no processing in S202 to S205 is performed, the same compositearea as that of all the frames can be used. Alternatively, a scenechange can be detected between frames, and, if there is a scene change,the processing in S202 to S205 can be performed.

In S216, the image processor 124 applies recording processing to thecomposite frame generated in S214. For example, the recording processingincludes file header generation and coding. The image processor 124records an image file including the composite frame in the recordingmedium 104 via the memory controller 115 and the recording medium I/F118.

In the moving image mode, the methods of calculating the evaluationvalues can be changed. For example, the evaluation value E can becalculated using the following equation:

E=Wy×Ey+Wu×Eu+Wv×Ev+Wd×Ed+Wf

Since the evaluation values other than the weight Wf are the same asthose in the still image mode, descriptions thereof are omitted. Theweight Wf is used for making a candidate area corresponding to acomposite area determined in the previous frame more likely to bedetermined as a composite area. This is because consecutive moving imageframes are highly correlated with each other and are highly likely tohave the same composite area, and this is also to prevent the compositearea from frequently moving. In this manner, advantageous effects thatare the same as or similar to those in the still image mode can beachieved in the moving image mode.

OTHER EMBODIMENTS

In the above-described embodiment, a composite area is determined fromamong candidate areas based on an evaluation value regarding at leastone of luminance information, color information, or distanceinformation. Alternatively, a composite area can be determined basedonly on distance information. For example, among candidate areas, acandidate area that has distance information (such as the averagedistance or the maximum value) that is greater than a predeterminedfirst threshold and that is the largest can be determined as a compositearea. Alternatively, among candidate areas, a candidate area that hasdistance information (such as the average distance or the maximum value)that is less than a predetermined second threshold and that is thesmallest can be determined as a composite area. In doing so, an area ofa very distant subject or a very close subject can be determined as acomposite area. The first threshold can be made greater than or equal tothe second threshold (first threshold second threshold).

In addition to distance information of candidate areas, distanceinformation of a prohibited area can also be taken into consideration.For example, a candidate area whose distance information (such as theaverage distance or the maximum value) has the greatest difference fromdistance information (such as the distance average value or the minimumvalue) of a prohibited area or a subject area can be determined as acomposite area.

For example, in the case of the scene illustrated in FIGS. 3A to 3D, abackground area whose distance information has a great difference fromdistance information of a prohibited area set in the center of the image(FIG. 3C) or the subject area and its surrounding areas (FIG. 3D) has alower importance level. If the center of the image is not a prohibitedarea as in FIG. 3D, the difference in distance information between abuilding at the center of the image and the prohibited area 301 is muchsmaller than the difference in distance information between theprohibited area 301 and the background. Thus, a building area isprevented from being determined as a composite area.

Alternatively, a candidate area whose distance information (such as theaverage distance or the minimum value) is not only greater, but alsosmaller than distance information of a prohibited area or a subjectarea, and has the greatest difference can be determined as a compositearea.

Although the above-described embodiment has discussed the case ofselecting a candidate area with the minimum or maximum evaluation valueor distance information, the advantageous effects are achievable whenthe evaluation value or distance information is not necessarily theminimum or maximum. Thus, one of candidate areas whose evaluation valueor distance information is smaller than a threshold can be selected, orone of candidate areas whose evaluation value or distance information isgreater than a threshold can be selected. If there are correspondingcandidate areas, a condition for selecting one area can be arbitrarilyset. It can be configured that the user selects one of candidate areas.

Embodiments can also be realized by a process of supplying a programrealizing one or more functions of the above-described embodiment to asystem or an apparatus via a network or a storage medium, and readingand executing the program with one or more processors included in acomputer of the system or apparatus. Embodiments can also be realized bya circuit (such as an application-specific integrated circuit (ASIC))that realizes one or more functions.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

According to embodiments of the invention, an image processing apparatusand an image processing method that determines an area to be combinedwith an image, regardless of the presence of the face of a person can beprovided.

While exemplary embodiments have been described, it is to be understoodthat the invention is not limited to the disclosed exemplaryembodiments. The scope of the following claims is to be accorded thebroadest interpretation so as to encompass all such modifications andequivalent structures and functions.

This application claims the benefit of Japanese Patent Application No.2015-225085, filed Nov. 17, 2015, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus comprising: one ormore processors; and a memory storing instructions which, when theinstructions are executed by the one or more processors, cause the imageprocessing apparatus to function as: an obtaining unit configured toobtain a first image and a second image; a determination unit configuredto determine a partial area of the first image as a composite area to becombined with the second image; and a combining unit configured tocombine the second image with the composite area, wherein thedetermination unit determines the composite area based on distanceinformation with regard to a plurality of partial areas of the firstimage.
 2. The image processing apparatus according to claim 1, whereinthe determination unit further sets a prohibited area in the firstimage, and does not set, among the plurality of partial areas, a partialarea that overlaps the prohibited area as the composite area.
 3. Theimage processing apparatus according to claim 2, wherein the prohibitedarea is at least one of a central area of the first image or aparticular subject area and its surrounding areas included in the firstimage.
 4. The image processing apparatus according to claim 2, whereinthe determination unit determines the composite area from, among theplurality of partial areas, partial areas whose distance information hasa difference from distance information of the prohibited area that isgreater than a threshold.
 5. The image processing apparatus according toclaim 1, wherein the determination unit determines the composite areafrom, among the plurality of partial areas, partial areas whose distanceinformation is greater than a predetermined threshold.
 6. The imageprocessing apparatus according to claim 1, wherein the determinationunit determines the composite area from, among the plurality of partialareas, partial areas whose distance information is less than apredetermined threshold.
 7. The image processing apparatus according toclaim 1, wherein the determination unit determines the composite areafrom, among the plurality of partial areas, partial areas whosevariation regarding the distance information is less than a threshold.8. The image processing apparatus according to claim 1, wherein thedistance information is information indicating a subject distance inunits of pixels of the first image.
 9. An image processing apparatuscomprising: one or more processors; and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the image processing apparatus to function as: an obtaining unitconfigured to obtain a first image and a second image; a determinationunit configured to determine a partial area of the first image as acomposite area to be combined with the second image; and a combiningunit configured to combine the second image with the composite area,wherein the determination unit determines the composite area from, amonga plurality of partial areas of the first image, partial areas whosevariation regarding at least one of luminance information and colorinformation is less than a threshold.
 10. An image capturing apparatuscomprising: an image sensor; a generator configured to generate a firstimage and a second image from an image obtained by the image sensor; andan image processing apparatus comprising: one or more processors; and amemory storing instructions which, when the instructions are executed bythe one or more processors, cause the image processing apparatus tofunction as: an obtaining unit configured to obtain a first image and asecond image; a determination unit configured to determine a partialarea of the first image as a composite area to be combined with thesecond image; and a combining unit configured to combine the secondimage with the composite area, wherein the determination unit determinesthe composite area based on distance information with regard to aplurality of partial areas of the first image.
 11. An image processingmethod executed by an image processing apparatus, comprising: obtaininga first image and a second image; determining a partial area of thefirst image as a composite area to be combined with the second imagebased on distance information with regard to a plurality of partialareas of the first image; and combining the second image with thecomposite area.
 12. A non-transitory computer-readable storage mediumstoring computer executable instructions for causing a computer toexecute an image processing method by an image processing apparatus, theimage processing method comprising: obtaining a first image and a secondimage; determining a partial area of the first image as a composite areato be combined with the second image based on distance information withregard to a plurality of partial areas of the first image; and combiningthe second image with the composite area.